{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基于scikit-surprise构建推荐系统\n",
    "- surprise 包含近邻算法（协同过滤），基于矩阵分解（隐语义模型、NMF等）的方法等的推荐应用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在推荐系统的建模过程中，我们将用到python库 [Surprise(Simple Python RecommendatIon System Engine)](https://github.com/NicolasHug/Surprise)，是scikit系列中的一个(很多同学用过scikit-learn和scikit-image等库)。\n",
    "\n",
    "### 简单易用，同时支持多种推荐算法：\n",
    "* [基础算法/baseline algorithms](http://surprise.readthedocs.io/en/stable/basic_algorithms.html)\n",
    "* [基于近邻方法(协同过滤)/neighborhood methods](http://surprise.readthedocs.io/en/stable/knn_inspired.html)\n",
    "* [矩阵分解方法/matrix factorization-based (SVD, PMF, SVD++, NMF)](http://surprise.readthedocs.io/en/stable/matrix_factorization.html#surprise.prediction_algorithms.matrix_factorization.SVD)\n",
    "\n",
    "| 算法类名        | 说明  |\n",
    "| ------------- |:-----|\n",
    "|[random_pred.NormalPredictor](http://surprise.readthedocs.io/en/stable/basic_algorithms.html#surprise.prediction_algorithms.random_pred.NormalPredictor)|Algorithm predicting a random rating based on the distribution of the training set, which is assumed to be normal.|\n",
    "|[baseline_only.BaselineOnly](http://surprise.readthedocs.io/en/stable/basic_algorithms.html#surprise.prediction_algorithms.baseline_only.BaselineOnly)|Algorithm predicting the baseline estimate for given user and item.|\n",
    "|[knns.KNNBasic](http://surprise.readthedocs.io/en/stable/knn_inspired.html#surprise.prediction_algorithms.knns.KNNBasic)|A basic collaborative filtering algorithm.|\n",
    "|[knns.KNNWithMeans](http://surprise.readthedocs.io/en/stable/knn_inspired.html#surprise.prediction_algorithms.knns.KNNWithMeans)|A basic collaborative filtering algorithm, taking into account the mean ratings of each user.|\n",
    "|[knns.KNNBaseline](http://surprise.readthedocs.io/en/stable/knn_inspired.html#surprise.prediction_algorithms.knns.KNNBaseline)|A basic collaborative filtering algorithm taking into account a baseline rating.|\t\n",
    "|[matrix_factorization.SVD](http://surprise.readthedocs.io/en/stable/matrix_factorization.html#surprise.prediction_algorithms.matrix_factorization.SVD)|The famous SVD algorithm, as popularized by Simon Funk during the Netflix Prize.|\n",
    "|[matrix_factorization.SVDpp](http://surprise.readthedocs.io/en/stable/matrix_factorization.html#surprise.prediction_algorithms.matrix_factorization.SVDpp)|The SVD++ algorithm, an extension of SVD taking into account implicit ratings.|\n",
    "|[matrix_factorization.NMF](http://surprise.readthedocs.io/en/stable/matrix_factorization.html#surprise.prediction_algorithms.matrix_factorization.NMF)|A collaborative filtering algorithm based on Non-negative Matrix Factorization.|\n",
    "|[slope_one.SlopeOne](http://surprise.readthedocs.io/en/stable/slope_one.html#surprise.prediction_algorithms.slope_one.SlopeOne)|A simple yet accurate collaborative filtering algorithm.|\n",
    "|[co_clustering.CoClustering](http://surprise.readthedocs.io/en/stable/co_clustering.html#surprise.prediction_algorithms.co_clustering.CoClustering)|A collaborative filtering algorithm based on co-clustering.|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 其中基于近邻的方法(协同过滤)可以设定不同的度量准则。\n",
    "\n",
    "| 相似度度量标准 | 度量标准说明  |\n",
    "| ------------- |:-----|\n",
    "|[cosine](http://surprise.readthedocs.io/en/stable/similarities.html#surprise.similarities.cosine)|Compute the cosine similarity between all pairs of users (or items).|\n",
    "|[msd](http://surprise.readthedocs.io/en/stable/similarities.html#surprise.similarities.msd)|Compute the Mean Squared Difference similarity between all pairs of users (or items).|\n",
    "|[pearson](http://surprise.readthedocs.io/en/stable/similarities.html#surprise.similarities.pearson)|Compute the Pearson correlation coefficient between all pairs of users (or items).|\n",
    "|[pearson_baseline](http://surprise.readthedocs.io/en/stable/similarities.html#surprise.similarities.pearson_baseline)|Compute the (shrunk) Pearson correlation coefficient between all pairs of users (or items) using baselines for centering instead of means.|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 支持不同的评估准则\n",
    "| 评估准则 | 准则说明  |\n",
    "| ------------- |:-----|\n",
    "|[rmse](http://surprise.readthedocs.io/en/stable/accuracy.html#surprise.accuracy.rmse)|Compute RMSE (Root Mean Squared Error).|\n",
    "|[msd](http://surprise.readthedocs.io/en/stable/similarities.html#surprise.similarities.msd)|Compute MAE (Mean Absolute Error).|\n",
    "|[fcp](http://surprise.readthedocs.io/en/stable/accuracy.html#surprise.accuracy.fcp)|Compute FCP (Fraction of Concordant Pairs).|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### surprise中有三种不同的数据类：\n",
    "- Dataset：可以直接或通过交叉验证迭代器拆分为训练集和测试集。后者意味着如果你在交叉验证中将一个Dataset作为参数传递，它将创建许多训练-测试拆分。\n",
    "- Trainset：在模型的fit方法中用作参数。\n",
    "- Testset：在模型的test方法中用作参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 入门实例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating RMSE, MAE of algorithm SVD on 5 split(s).\n",
      "\n",
      "                  Fold 1  Fold 2  Fold 3  Fold 4  Fold 5  Mean    Std     \n",
      "RMSE (testset)    0.9310  0.9362  0.9331  0.9319  0.9444  0.9353  0.0049  \n",
      "MAE (testset)     0.7314  0.7415  0.7348  0.7356  0.7442  0.7375  0.0047  \n",
      "Fit time          3.22    3.26    3.28    3.30    3.26    3.26    0.02    \n",
      "Test time         0.09    0.09    0.09    0.10    0.09    0.09    0.00    \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'test_rmse': array([0.93095971, 0.93623746, 0.93309525, 0.93185646, 0.94435713]),\n",
       " 'test_mae': array([0.73143468, 0.74146104, 0.73482802, 0.73564954, 0.74422306]),\n",
       " 'fit_time': (3.2247302532196045,\n",
       "  3.25573468208313,\n",
       "  3.279620885848999,\n",
       "  3.2954795360565186,\n",
       "  3.25848650932312),\n",
       " 'test_time': (0.09175705909729004,\n",
       "  0.09175443649291992,\n",
       "  0.09075331687927246,\n",
       "  0.09589624404907227,\n",
       "  0.08975958824157715)}"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 引入SVD矩阵分解\n",
    "from surprise import SVD\n",
    "from surprise import Dataset\n",
    "# 引入交叉验证\n",
    "from surprise.model_selection import cross_validate\n",
    "\n",
    "data = Dataset.load_builtin('ml-100k')\n",
    "\n",
    "algo = SVD()\n",
    "# cv=5表示5折交叉验证\n",
    "cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=5, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 示例一：基于近邻算法构建模型并进行预测"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.基于物品的协同过滤\n",
    "- 以下的程序段告诉大家如何在协同过滤算法建模以后，根据一个item取回相似度最高的item，主要是用到algo.get_neighbors()这个函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import io\n",
    "# 引入KNN近邻的协同过滤算法\n",
    "from surprise import KNNBaseline, Reader\n",
    "from surprise import Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_path = os.path.expanduser('./ratings.dat')\n",
    "reader = Reader(line_format='user item rating timestamp', sep='::')\n",
    "movie_data = Dataset.load_from_file(file_path, reader=reader)\n",
    "# 使用所有评分来构建一个Trainset对象：\n",
    "trainset = movie_data.build_full_trainset()\n",
    "# 设置相似性度量准则\n",
    "sim_options = {'name': 'pearson_baseline', 'user_based': False}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "电影数量 => 28682\n",
      "观众数量 => 50201\n"
     ]
    }
   ],
   "source": [
    "print('电影数量 =>', trainset.n_items)\n",
    "print('观众数量 =>', trainset.n_users)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获取电影名到电影ID的映射 + 电影ID到电影名的映射\n",
    "def read_item_names():\n",
    "    file_name = os.path.expanduser('./movies.dat')\n",
    "    rid_to_name = {}\n",
    "    name_to_rid = {}\n",
    "    with io.open(file_name, 'r', encoding='ISO-8859-1') as f:\n",
    "        for line in f:\n",
    "            line = line.split('::')\n",
    "            rid_to_name[line[0]] = line[1]\n",
    "            name_to_rid[line[1]] = line[0]\n",
    "    return rid_to_name, name_to_rid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimating biases using als...\n",
      "Computing the pearson_baseline similarity matrix...\n",
      "Done computing similarity matrix.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<surprise.prediction_algorithms.knns.KNNBaseline at 0x24915c5a7f0>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 计算相似度\n",
    "alg = KNNBaseline(sim_options=sim_options)\n",
    "alg.fit(trainset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "movie_raw_id => 1790809\n",
      "movie_inner_id => 1290\n"
     ]
    }
   ],
   "source": [
    "# 获取电影名到电影id 和 电影id到电影名的映射\n",
    "rid_to_name, name_to_rid = read_item_names()\n",
    "# 拿出来Pirates of the Caribbean: Dead Men Tell No Tales这部电影对应的item id\n",
    "movie_raw_id = name_to_rid[\"Pirates of the Caribbean: Dead Men Tell No Tales (2017)\"]\n",
    "movie_inner_id = alg.trainset.to_inner_iid(movie_raw_id)\n",
    "print('movie_raw_id =>', movie_raw_id)\n",
    "print('movie_inner_id =>', movie_inner_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[761, 1700, 11, 793, 71, 1030, 69, 347, 3223, 762]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 找到电影Pirates of the Caribbean: Dead Men Tell No Tales最近的10个邻居\n",
    "movie_neighbors = alg.get_neighbors(movie_inner_id, k=10)\n",
    "movie_neighbors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The 10 nearest neighbors of <Pirates of the Caribbean: Dead Men Tell No Tales (2017)> are:\n",
      "X-Men: Apocalypse (2016)\n",
      "The Fate of the Furious (2017)\n",
      "Deadpool (2016)\n",
      "Thor: The Dark World (2013)\n",
      "The Expendables 3 (2014)\n",
      "Passengers (2016)\n",
      "Transcendence (2014)\n",
      "Deepwater Horizon (2016)\n",
      "Night at the Museum: Secret of the Tomb (2014)\n",
      "Jack Reacher: Never Go Back (2016)\n"
     ]
    }
   ],
   "source": [
    "# 从近邻的ID映射回电影名称\n",
    "movie_neighbors_raw_ids = []\n",
    "for inner_id in movie_neighbors:\n",
    "    movie_neighbors_raw_ids.append(alg.trainset.to_raw_iid(inner_id))\n",
    "\n",
    "movie_names = []\n",
    "for rid in movie_neighbors_raw_ids:\n",
    "    movie_names.append(rid_to_name[rid])\n",
    "    \n",
    "print(\"The 10 nearest neighbors of <Pirates of the Caribbean: Dead Men Tell No Tales (2017)> are:\")\n",
    "for movie in movie_names:\n",
    "    print(movie)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 对用户进行推荐\n",
    "- 查看指定用户的观影历史，推荐和观看过的电影相似度很高的电影进行推荐"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "获取用户inner_id为1000的电影打分字典集合 => [(369, 9.0), (1778, 9.0), (3389, 10.0)]\n"
     ]
    }
   ],
   "source": [
    "user_inner_id = 1000\n",
    "user_rating = trainset.ur[user_inner_id]\n",
    "print('获取用户inner_id为1000的电影打分字典集合 =>', user_rating)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "当前电影编号 => 369\n",
      "user: 1000       item: 369        r_ui = 5.00   est = 5.00   {'was_impossible': False} It's a Wonderful Life (1946)\n",
      "当前电影编号 => 1778\n",
      "user: 1000       item: 1778       r_ui = 5.00   est = 5.00   {'was_impossible': False} The Thinning (2016)\n",
      "当前电影编号 => 3389\n",
      "user: 1000       item: 3389       r_ui = 5.00   est = 5.00   {'was_impossible': False} Wasati (2016)\n"
     ]
    }
   ],
   "source": [
    "items = map(lambda x:x[0], user_rating)\n",
    "for movie in items:\n",
    "    print('当前电影编号 =>', movie)\n",
    "    print(alg.predict(user_inner_id, movie, r_ui=5), rid_to_name[alg.trainset.to_raw_iid(movie)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 示例二：基于矩阵分解构建模型并进行预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from surprise import SVDpp\n",
    "from surprise import Dataset\n",
    "from surprise.model_selection import GridSearchCV\n",
    "\n",
    "# 网格搜索进行参数调优\n",
    "param_grid = {'n_epochs': [5, 10], 'lr_all': [0.002, 0.005],\n",
    "              'reg_all': [0.4, 0.6]}\n",
    "gs = GridSearchCV(SVDpp, param_grid, measures=['rmse', 'mae'], cv=3, n_jobs=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "# 训练模型，使用网格搜索进行参数调优的时候，训练数据类型需要为dataset类型\n",
    "gs.fit(movie_data)\n",
    "\n",
    "print('total time =>', (time.time() - start), 'seconds!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
